Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

breaking: allowmean and allowdrift True by default in AutoARIMA #918

Merged
merged 5 commits into from
Sep 20, 2024
Merged

breaking: allowmean and allowdrift True by default in AutoARIMA #918

merged 5 commits into from
Sep 20, 2024

Conversation

V-nguard
Copy link
Contributor

Hello,

I encountered the same issue as #896, this should fix it.
I couldn't commit without adding the other files; I don't think I missed a step in the contributing guidelines either.

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@CLAassistant
Copy link

CLAassistant commented Sep 19, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@jmoralez
Copy link
Member

Hey. Thanks for your contribution. Can you please add the pin nbdev==2.3.25 to the settings.ini file and restore the other files? Seems like that was added in v2.3.28

@V-nguard
Copy link
Contributor Author

Hi, I'm not fully fluent in git, so is there another way to restore the files without creating another commit? Not sure what the intended operations are (like I'm assuming I don't commit the settings.ini change either because that should be another pull request)

@jmoralez
Copy link
Member

I think you can run git checkout main -- python to restore all the python files and then nbdev_export again.

It's ok to commit the settings.ini file here, it only changes a dev requirement.

@V-nguard
Copy link
Contributor Author

Thanks for the git command. This should work.

@jmoralez jmoralez linked an issue Sep 19, 2024 that may be closed by this pull request
[AutoARIMA] Incorrect default parameters in AutoARIMA #896
Closed
@jmoralez jmoralez changed the title fix: AutoARIMA docstring and constructor mismatch for allowdrift and … breaking: allowmean and allowdrift True by default in AutoARIMA Sep 19, 2024
@jmoralez
Copy link
Member

@V-nguard seems like this improves the performance on the M3 dataset. Can you please set this line

statsforecast/experiments/m3/src/evaluation.py

Line 53 in 8e16c02

'AutoARIMA': 4.57,
to 4.46 (logs)?

@V-nguard
Copy link
Contributor Author

I did the change, but I can't commit due to ruff. Something about imported numpy but unused. I ran ruff check --fix *, and now it's saying

experiments/m3/src/evaluation.py:41: error: No overload variant of "__getitem__" of "list" matches argument type "list[str]"  [call-overload]
experiments/m3/src/evaluation.py:41: note: Possible overload variants:
experiments/m3/src/evaluation.py:41: note:     def __getitem__(self, SupportsIndex, /) -> Any
experiments/m3/src/evaluation.py:41: note:     def __getitem__(self, slice, /) -> list[Any]
experiments/m3/src/evaluation.py:42: error: No overload variant of "__getitem__" of "list" matches argument type "str"  [call-overload]
experiments/m3/src/evaluation.py:42: note: Possible overload variants:
experiments/m3/src/evaluation.py:42: note:     def __getitem__(self, SupportsIndex, /) -> Any
experiments/m3/src/evaluation.py:42: note:     def __getitem__(self, slice, /) -> list[Any]
experiments/m3/src/evaluation.py:42: error: No overload variant of "__setitem__" of "list" matches argument types "str", "Any"  [call-overload]
experiments/m3/src/evaluation.py:42: note:     def __setitem__(self, SupportsIndex, Any, /) -> None
experiments/m3/src/evaluation.py:42: note:     def __setitem__(self, slice, Iterable[Any], /) -> None
experiments/m3/src/evaluation.py:43: error: "list[Any]" has no attribute "set_index"  [attr-defined]
experiments/m3/src/evaluation.py:44: error: "list[Any]" has no attribute "columns"  [attr-defined]
experiments/m3/src/evaluation.py:45: error: "list[Any]" has no attribute "set_index"  [attr-defined]
experiments/m3/src/evaluation.py:46: error: "list[Any]" has no attribute "droplevel"  [attr-defined]
experiments/m3/src/evaluation.py:47: error: "list[Any]" has no attribute "to_csv"  [attr-defined]
experiments/m3/src/evaluation.py:48: error: "list[Any]" has no attribute "query"  [attr-defined]
experiments/m3/src/evaluation.py:49: error: "list[Any]" has no attribute "query"  [attr-defined]
experiments/m3/src/evaluation.py:60: error: "dict[str, float]" has no attribute "index"  [attr-defined]
Found 11 errors in 1 file (checked 3 source files)

@jmoralez
Copy link
Member

Yeah, sorry that's mypy. We don't really care about running it on the experiments folder, so can you change this

statsforecast/.pre-commit-config.yaml

Line 28 in 8e16c02

exclude: 'setup.py'

to (experiments|setup.py)?

@V-nguard
Copy link
Contributor Author

Done!

@jmoralez
Copy link
Member

Hmm, it seems that the smape for the statistical ensemble in the m3 is increasing in the test, however I'm not able to reproduce that score locally, I've already tried on two different machines and I get what we currently have. I'll keep investigating.

@jmoralez
Copy link
Member

Oh it was due to me having the NIXTLA_ID_AS_COL env variable set, which considered the index when creating the ensemble. I'm able to reproduce the score on the test now, can you please update that one as well? i.e. set this:

statsforecast/experiments/m3/src/evaluation.py

Line 57 in 8e16c02

'StatisticalEnsemble': 4.23,
to 4.3.

jmoralez
jmoralez approved these changes Sep 20, 2024
View reviewed changes
Copy link
Member

@jmoralez jmoralez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@jmoralez jmoralez requested a review from AzulGarza September 20, 2024 02:48
AzulGarza
AzulGarza approved these changes Sep 20, 2024
View reviewed changes
Copy link
Member

@AzulGarza AzulGarza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome @V-nguard and @jmoralez! lgtm

@jmoralez jmoralez merged commit 13c0b35 into Nixtla:main Sep 20, 2024
30 checks passed
@jmoralez jmoralez mentioned this pull request Sep 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AzulGarza AzulGarza AzulGarza approved these changes

@jmoralez jmoralez jmoralez approved these changes

Assignees
No one assigned
Labels
breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[AutoARIMA] Incorrect default parameters in AutoARIMA
4 participants
@V-nguard @CLAassistant @jmoralez @AzulGarza